Overview

When requesting an event from a store or a semaphore it is often necessary to request not just the next available read buffer but some specific read buffer. In order to achieve this, CLIP introduces the notion of keys and rules, which allow individual buffers to be tagged with a value that is interrogated when they are requested.

Data Store Static/Dynamic Keys

Normally store writer keys are statically set when the store is opened, however, keys are not actually applied until the store record is constructed. Thus a writer can open a store with the default key, and dynamically set a new key later when the store's buffer is constructed (requires an overloaded Construct() to be defined). This allows an active object to grab a write buffer immediately, but delay setting the key until it has made a separate decision based on some programmatic criteria.

Passive objects (event propagators) always open stores with static keys. If a dynamic key is required the passive objects should use the default key and the active object writers can then set the key dynamically when they construct the store buffer.

Active object store readers can apply dynamic keys if they make a direct auxiliary connection to the store. Any connections made via passive objects or trigger connections use static keys. In this case, readers can only 'filter' keys by opening the store with the default key and then checking what key was actually specified. If an invalid key is found the store can be closed/aborted. However, care must be used in deciding which to do (See Transient Stores - Aborting).

Element Numbers as Keys

When specifying object element numbers for keys on connections (In the Properties window for a connection) use elems.ElemNum(). NB. this is the element number of the object doing the connecting/opening, not the element number of the object being connected to. Macros can also be used.

However, when using <>Cxn.OpenRead/Write() functions in Methods/Threads use the ElemNum() function (or this->ElemNum() ).

Examples

Prioritisation

Keys and rules can be used to implement a priority queue using a transient store. When writing into the store, a key is applied with its value set to the priority of the buffer. When reading out the MAX rule is used to read the buffer with the highest priority first.

Routing

Keys can be used to route particular buffers to particular consumers. When writing to the store, the target 'address' is set as the buffer key. When reading out, the EQUAL rule is used with the 'address' of the consumer. Consider the following example of a single writer and three readers:

The writer sets a key value of 0, 1 or 2 depending on which reader it the data is intended for. The reading method uses a key equal to its element number (element 0 requests with key 0 etc.). This allows the writer to route the data to whichever reader it requires.

Stream Reordering

This example circuit implements a simple scalable processing farm which has a maximum concurrency of 'NP', where 'NP' is the number of CPU cores discovered at runtime. It has a thread that reads a stream of tasks from file. These are then processed in parallel and the results written to an output file in the same order that the tasks were read. In the general case, the time to process a task is non-deterministic (execution time depends on it's data) and so completion order is unknown. This example uses keys and rules to ensure that the input and output order are consistent.

The following code is executed by the 'input' thread;

TASK nextTask;
Int  taskKey = 0;

while( TRUE )
{
   // Read each task from file
   if ( TaskFileDev().Read( nextTask ) )
   {
      // Open the Task store for write with key = 'taskKey'
      Aux1_TaskTst1Cxn().OpenWrite( taskKey, CLP_WAIT );

      // Construct the Task store record
      Aux1_TaskTst1Cxn().Record().Construct();

      // Copy the task to the store record
      Aux1_TaskTst1Cxn().Record().Data() = nextTask;

      // Close the store
      Aux1_TaskTst1Cxn().Close();

      // Increment the key and go around again
      taskKey++;
   }
   else
      ClpSleep(CLP_WAIT); // Stop when end-of-file reached
}

The 'Proc' processing methods compete for tasks from the task stream and write their output using the task's associated key. Note that output needs to be collected before input otherwise a task could be accepted by a method which could then find that its output was full; resulting in deadlock.

Int taskKey;

// Retrieve the task's key
taskKey = Trigger_TaskTst2Cxn().key();

// This method is passed an 'open' results store but the
// record needs to be constructed and passed the key
Trigger_ResultTst1Cxn().Record().Construct( taskKey );

// Now we can pass references to the task and results data
// objects for calculation
Calculate( Trigger_TaskTst2Cxn().Record().Data(),
Trigger_ResultTst1Cxn().Record().Data() );

// Now we can return and let the result propagate to
// the output thread
return;

Finally, the 'output' thread reads results using an incrementing key, which ensures that results are written to file in the same order that their corresponding tasks were read from file, regardless of the order that their processing completes in.

Uns taskKey = 0;

while( TRUE )
{
   // Wait for next result
   Aux1_ResultTst2().OpenRead( taskKey, CLP_EQ_RULE, CLP_WAIT );

   // Write result record to file
   ResultFileDev().Write( Aux1_ResultTst2().Record().Data() );

   // Increment key and go around again
   ++taskKey;
}

Notes

When writing to stores/semaphores with keys, developers need to ensure that all possible key/rule combinations are covered by Readers. Any keyed buffers that are not read will ultimately fill the store. A generic catch all "No Rule" connection (akin to the C++ switch default case) should not be used as the "No Rule" connection could read the event first, stopping it going to the intended recipient. If a generic catch all-others option is required a non-overlapping "Greater Than/Less Than/Not_Equal" rule could be used.

Blueprint Help	Send comments on this topic.
Keys and Rules